UAB-FT-491 

KA-TP-22-2000 

hep-ph/0011091 



Loop Induced Flavor Changing Neutral Decays of the Top Quark 
in a General Two- Higgs- Doublet Model 



Santi Bejar", Jaume Guasch'', 
Joan Sola" 



"'Grup de Fisica Tedrica and Institut de Fisica d'Altes Energies, 
Universitat Autononia de Barcelona, E-08193, Bellaterra, Barcelona, Catalonia, Spain 
^Institut fiir Theoretische Physik, Universitat Karlsruhe, KaiserstraBe 12, D-76128 

Karlsruhe, Germany. 



ABSTRACT 

Decays of the top quark induced by flavor changing neutral currents (FCNC) 
are known to be extremely rare events within the Standard Model. This is 
so not only for the decay modes into gauge bosons, but most notably in the 
case of the Higgs channels, e.g. t Hsm + c, with a branching fraction of 
10~^^ at most. Therefore, detection of FCNC top quark decays in a future 
high-energy, and high-luminosity, machine like the LHC or the LC would be 
an indisputable signal of new physics. In this paper we show that within the 
simplest extension of the SM, namely the general two-Higgs-doublet model, 
the FCNC top quark decays into Higgs bosons, t — > {h^, if °, A^) + c, can be 
the most favored FCNC modes - comparable or even more efficient than the 
gluon channel t g + c. In both cases the optimal results are obtained for 
Type II models. However, only the Higgs channels can have rates reaching the 
detectable level (10~^), with a maximum of order lO""^ which is compatible 
with the charged Higgs bounds from radiative B-meson decays. We compare 
with the previous results obtained in the Higgs sector of the MSSM. 



1 Introduction 



In the near and middle future, with the upgrades of the Tevatron (Run II, TeV33), the 
advent of the LHC, and the construction of an e^e~ linear collider (LC), new results on 
top quark physics ^ , and possibly also on Higgs physics, will be obtained that may be 
extremely helpful to complement the precious information already collected at LEP 100 
and 200 from Z and W physics. Both types of machines, the hadron colliders and the LC 
will work at high luminosities and will produce large amounts of top quarks. In the LHC, 
for example, the production of top quark pairs will be a{tt) = 800 pb - roughly two orders 
of magnitude larger than in the Tevatron Run II. In the so-called low-luminosity phase 
(10^^ cm~^s~^) of the LHC one expects about three tt-pair per second (ten million ti- 
pairs per year!) And this number will be augmented by one order of magnitude in the 
high-luminosity phase (10^^ cm~^s~^). As for a future LC running at e.g. ^/s = 500 GeV, 
one has a smaller cross-section cr{tt) = 650 fb but a higher luminosity factor ranging from 
5 X 10^^ cm~^s~^ to 5 X 10^^ cm~^s^^ and of course a much cleaner environment With 
datasets from LHC and LC increasing to several 100/6~^/year in the high-luminosity 
phase, one should be able to pile up an enormous wealth of statistics on top quark decays. 
Therefore, not surprisingly, these machines should be very useful to analyze rare decays 
of the top quark, viz. decays whose branching fractions are so small (< 10^^) that they 
could not be seen unless the number of collected top decays is very large. 

The reason for the interest in these decays is at least twofold. First, the typical 
branching ratios for the rare top quark decays predicted within the Standard Model (SM) 
are so small that the observation of a single event of this kind should be "instant evidence" , 
so to speak, of new physics; and second, due to its large mass (m^ = 174.3 ± 5.1 GeV |^), 
the top quark could play a momentous role in the search for Higgs physics beyond the 
SM. While this has been shown to be the case for the top quark decay modes into charged 
Higgs bosons, both in the Minimal Supersymmetric Standard Model (MSSM) and in a 
general two-Higgs-doublet model (2HDM) [|, HQ we expect that a similar situation could 
apply for top quark decays into non-SM neutral Higgs bosons. Notice that the latter are 
rare top quark events of a particularly important kind: they are decays of the top quark 
mediated by flavor changing neutral currents (FCNC). 

At the tree-level there are no FCNC processes in the SM, and at one-loop they are 
induced by charged-current interactions, which are GIM-suppressed . In particular, 
FCNC decays of the top quark into gauge bosons {t cV; V = •j, Z, g) are very unlikely. 
For the present narrow range of masses for the top quark, they yield maximum branching 
ratios of ~ 5 x 10^^'^ for the photon, slightly above 1 x 10~^^ for the Z-boson, and 
~ 4 X 10^^^ for the gluon channel]^]. These are much smaller than the FCNC rates of 
a typical low-energy meson decay, e.g. B[b — >■ 57) ~ 10~^. And the reason is simple: 
for FCNC top quark decays in the SM, the loop amplitudes are controlled by down-type 
quarks, mainly by the bottom quark. Therefore, the scale of the loop amplitudes is set 
by ml and the partial widths are of order 
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where a is a em for = 7, Z and as^oiV = g. The factor F ~ {l — rriy/mlY results, upon 
neglecting rric, from phase space and polarization sums. Notice that the dimensionless 
fourth power mass ratio, in parenthesis in eq. (JlD, stems from the GIM mechanism and is 
responsible for the ultralarge suppression beyond naive expectations based on pure dimen- 
sional analysis, power counting and CKM matrix elements. From that simple formula, 
the approximate orders of magnitude mentioned above ensue immediately. 

Even more dramatic is the situation with the top quark decay into the SM Higgs boson, 
t ^ c HsMi which has recently been recognized to be much more disfavored than originally 
claimed 0: BR{t cHsm) ~ 10"^=^ - IQ-^^ {mt = 175 GeV; Mz < Mh < 2 Mw) JTU . 
This extremely tiny rate is far out of the range to be covered by any presently conceivable 
high luminosity machine. On the other hand, the highest FCNC top quark rate in the SM, 
namely that of the gluon channel t — > c(7, is still 6 orders of magnitude below the feasible 
experimental possibilities at the LHC. All in all the detection of FCNC decays of the 
top quark at visible levels (viz. BR{t cX) > 10~^) by the high luminosity colliders 
round the corner (especially LHC and LC) seems doomed to failure in the absence of 
new physics. Thus the possibility of large enhancements of some FCNC channels up 
to the visible threshold, particularly within the context of the general 2HDM and the 
MSSM, should be very welcome. Unfortunately, although the FCNC decay modes into 
electroweak gauge bosons Vew = W, Z may be enhanced a few orders of magnitude, it 
proves to be insufficient to raise the meager SM rates mentioned before up to detectable 
limits, and this is true both in the 2HDM - where BR{t —>■ Vewc) < 10^^ - and in 
the MSSM - where BR{t V^w c) < 10""^ except in highly unlikely regions of the MSSM 
parameter space [00- In this respect it is a lucky fact that these bad news need not to 
apply to the gluon channel, which could be barely visible {BR{t g c) ^ 10~^) both in 
the MSSM |T3| , |T^ and in the general 2HDM [^. But, most significant of all, they may 



not apply to the non-SM Higgs boson channels t —* {h^, H^, A^) + c either. As we shall 
show in the sequel, these Higgs decay channels of the top quark could lie above the visible 
threshold for a parameter choice made in perfectly sound regions of parameter space! 



While a systematic discussion of these "gifted" Higgs channels was made in Ref. [|14 
for the MSSM case and in other models^, to the best of our knowledge there is no similar 
study in the general 2HDM. And we believe that this study is necessary, not only to assess 
what are the chances to see traces of new (renormalizable) physics in the new colliders 
round the corner but also to clear up the nature of the virtual effects; in particular to dis- 
entangle whether the origin of the hypothetically detected FCNC decays of the top quark 
is ultimately triggered by SUSY or by some alternative, more generic, renormalizable 
extension of the SM such as the 2HDM or generalizations thereof. Of course the alleged 
signs of new physics could be searched for directly through particle tagging, if the new 
particles were not too heavy. However, even if accessible, the corresponding signatures 
could be far from transparent. In contrast, the indirect approach based on the FCNC 
processes has the advantage that one deals all the time with the dynamics of the top 



^Namely, regions in which there are wave-function renormaUzation thresholds due to (extremely for- 
tuitous!) sharp coincidences between the sum of the sparticle masses involved in the self-energy loops 
and the top quark mass. See e.g. Ref. for similar situations already in the conventional t Wb 
decay within the MSSM. In our opinion these narrow regions should not be taken too seriously. 

^Preliminary SUSY analysis of the Higgs channels are given in but they assume the MSSM Higgs 
mass relations at the tree-level. Therefore these are particular cases of the general MSSM approach given 
in [0. Studies beyond the MSSM (e.g. including R-parity violation) and also in quite different contexts 
from the present one are available in the literature, see e.g. [EgI. 
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Figure 1: One-loop vertex diagrams contributing to the FCNC top quark decays @). 
Shown are the vertices and mixed self-energies with all possible contributions from the SM 
fields and the Higgs bosons from the general 2HDM. The Goldstone boson contributions 
are computed in the Feynman gauge. 

quark. Thus by studying potentially new features beyond the well-known SM properties 
of this quark one can hopefully uncover the existence of the underlying new interactions. 



2 Relevant fields and interactions in the 2HDM 

We will mainly focus our interest on the loop induced FCNC decays 

t^ch {h = h^,H^,A^), (2) 

in which any of the three possible neutral Higgs bosons from a general 2HDM can be 
in the final state. However, as a reference we shall compare throughout our analysis the 
Higgs channels with the more conventional gluon channel 

t-^cg. (3) 

Although other quarks could participate in the final state of these processes, their contri- 
bution is obviously negligible - because it is further CKM-suppressed. The lowest order 
diagrams entering these decays are one-loop diagrams in which Higgs, quarks, gauge and 
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Goldstone bosons - in the Feynman gauge - circulate around. While the diagrams for 
the decays (H) are depicted in Fig. |l|, the ones for the decay are not shown Here 
we follow the standard notation namely h^, are CP-even Higgs bosons (where by 
convention m/jo < m//o) and is a CP-odd one (often called a "pseudoscalar" ) . As it is 
well-known, when the quark mass matrices are diagonalized in non-minimal extensions of 
the Higgs sector of the SM, the Yukawa couplings do not in general become simultaneously 
diagonalized, so that one would expect Higgs mediated FCNC's at the tree-level. These 
are of course unwanted, since they would lead to large FCNC processes in light quark 
phenomenology, which are stringently restricted by experiment. Apart from letting the 
Higgs masses to acquire very large values, one has two additional, more elegant, canonical 
choices to get rid of them. In Type I 2HDM (also denoted 2HDM I) one Higgs doublet, 
$1, does not couple to fermions at all and the other Higgs doublet, $2, couples to fermions 
in the same manner as in the SM. In contrast, in Type II 2HDM (also denoted 2HDM II) 
one Higgs doublet, $i, couples to down quarks (but not to up quarks) while $2 does the 
other way around. Such a coupling pattern is automatically realized in the framework 
of supersymmetry (SUSY), in particular in the MSSM, but it can also be arranged if we 
impose a discrete symmetry, e.g. $1 — >• — $1 and $2 +^2 (or vice versa) plus a suitable 
transformation for the right-handed quark fields. We shall not worry here on the ultimate 
theoretical origin of this ad hoc symmetry, but we will accept it as a guiding principle. As 
mentioned above, the SUSY case has recently been investigated in Ref. |T^, so we wish 



to concentrate here on Type I and Type II models of a sufficiently generic nature, to wit, 
those which are characterized by the following set of free parameters: 

(m/jo, m/^-o, m^o, m//±, tana, tan /5) , (4) 

where mH± is the mass of the charged Higgs companions if^, tana defines the mixing 
angle a in the diagonalization of the CP-even sector, and tan/? gives the mixing angle 
(5 in the CP-odd sector. The latter is a key parameter in our analysis. It is given by 
the quotient of the vacuum expectation values (VEV's) of the two Higgs doublets $2,1, 
viz. tan/? = V2/vi^ where the parameter sum v"^ = vl + f| is fixed by the W mass: 
= (l/2)(yf^t>^ (^f is the weak SU{2) gauge coupling) or, equivalently, by the Fermi 
constant: Gp = l/(2A/2)t>^. It is well-known Jl^ that the most general 2HDM Higgs 
potential subject to hermiticity, SU{2) x f/(l) gauge invariance and a discrete symmetry 
of the sort mentioned above involves six scalar operators with six free (real) coefficients 
Aj (i = 1, 2, . . . , 6) and the two VEV'sQ. We will furthermore assume that A5 = Ag in the 
general 2HDM Higgs potential This allows to remove the CP phase in the potential 
by redefining the phase of one of the doublets. In this way one can choose the VEV's of 
$1,2 real and positive. The alternative set (^ is just a (more physical) reformulation of 
this fact after diagonalization of the mass matrices and imposing the aforementioned set 
of constraints. 

As stated above, the two canonical types of 2HDM's only differ in the couplings to 
fermions but they share Feynman rules generally different from the corresponding ones 



^By imposing the discrete symmetry one is able, in principle, to get rid of two additional quartic and 
one bilinear operators in the Higgs potential. These quartics are not shown in , but the most general 
2HDM potential could contain all of these additional terms The bilinear ones are eventually kept 
in most cases as one usually makes allowance for the discrete symmetry to be only (softly) violated by 
the dimension two operators. The resulting model is still a minimal setup compatible with the absence 
of Higgs- mediated dangerous FCNC. 
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in the MSSM The necessary Feynman rules to compute the diagrams in Fig. |l| are 



mostly well-known. For instance, the interactions between Higgs and Goldstone bosons 
and the interactions between Goldstone bosons and gauge bosons are the same as in the 



MSSM • Similarly for the interactions among Goldstone bosons and gauge bosons, 
and Goldstone bosons and fermions. However, the 2HDM Feynman rules for the charged 
and neutral Higgs interactions with fermions and also for the trilinear self-couplings of 
Higgs bosons may drastically deviate from the MSSM. The charged Higgs interactions 
with fermions are encoded in the following interaction Lagrangian 

^%\b = H-h [mt cot (3PR + mb a,- Pl] t + h.c. (5) 

V 2 Mw 

where here, and hereafter, we use third-quark-family notation as a generic one; Vtb is 
the corresponding CKM matrix element, Pl,r = (1/2)(1 =F 75) are the chiral projection 
operators on left- and right-handed fermions, j = I, II runs over Type I and Type II 
2HDM's, and we have introduced a parameter aj such that aj = — cot (3 and aji = + tan /3. 
For the neutral Higgs interactions, the necessary pieces of the Lagrangian (see Fig. 0) 
can be written in the compact form 

_ -grrib 



hqq 



sin [3 

COS [3 



. COS a \ ^ I sm a 



sma cos a 



2Mw 



. . ^t\h° cosa + H'^ sina] t+ -—^ — —t-f^tA'^, (6) 

2 Mw sin /3 ^ ^ 2 Mw tan (3 ' ' ^ ^ 

where the upper row is for j = I and the down row is for j = II. As far as it goes to the 
2HDM Feynman rules for the trilinear couplings among Higgs bosons, and Higgses with 
Goldstone bosons, they are summarized in Table |l|, and are valid for Type I and Type II 
models. Let us recall that Type II models are specially important in that the Higgs sector 
of the MSSM is precisely of this sort. Had we not imposed the restriction A5 = Ag, then the 
trilinear rules would be explicitly dependent on the A5 parameter. However the numerical 
analysis that we perform in the next section does not depend in any essential way on this 
simplification. In essence we have just traded A5 for m^o in these rules and so by varying 
with respect to m^o we do explore most of the quantitative potential of the general 2HDM. 
In the MSSM case the condition A5 = Ag is automatic by the underlying supersymmetry, 
and the values of these couplings are determined by the SU{2) x f/(l) electroweak gauge 
couplings. But of course we treat here the parameters of Type II models in a way not 
restricted by SUSY prejudices. On the other hand there is no need to depart arbitrarily 
from the SUSY frame, as it can be useful for a better comparison. Be as it may, our 
Type II Higgs model is still sufficiently general that it cannot be considered as the limit 
of the MSSM when all the sparticle masses are decoupled. Both in the generic 2HDM II 
and in the MSSM, the Feynman rules for the lightest CP-even Higgs, go over to the 
SM Higgs boson ones in the limit sin(/? — a) 1. In the particular case of the MSSM, 
but not in a general 2HDM II, this limit is equivalent to rriAo —>■ 00 ■ Moreover, in the 
MSSM one hasQ ruho < 135 GeV whereas in the general Type II model there is no upper 
bound on rriho, and by the same token the corresponding lower bound is considerably less 
stringent (see below). 

^See PQl and references therein for the status of the Higgs mass calculations in the MSSM. 
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Table 1: Feynman rules for the trilinear couplings involving the Higgs self- interactions 
and the Higgs and Goldstone boson vertices in the Feynman gauge, with all momenta 
pointing inward. These rules are common to both Type I and Type II 2HDM under the 
conditions explained in the text. We have singled out some null entries associated to CP 
violation. The null vertices imply the corresponding deletion of some vertex diagrams in 
Fig. |1|. 

Since we shall perform our calculation in the on-shell scheme, we understand that the 
physical inputs are given by the electromagnetic coupling and the physical masses of all 
the particles: 

{e,Mw,Mz,mho,mHO,mAo,mH±,mf) . (7) 

The remaining parameters, except the Higgs mixing angles, are understood to be given 
in terms of the latter, e.g. the SU{2) gauge coupling appearing in the previous formulae 
and in Table ^ is given hj g = e/ , where the sinus of the weak mixing angle is defined 
through = 1 — M^/M|. It should be clear that, as there are no tree-level FCNC decays 
of the top quark, there is no need to introduce counterterms for the physical inputs in 
this calculation. In fact, the calculation is carried out in lowest order ("tree level") with 
respect to the effective tch and teg couplings and so the sum of all the one-loop diagrams 
(as well as of certain subsets of them) should be finite in a renormalizable theory, and 
indeed it is. 

3 Numerical analysis 

From the previous interaction Lagrangians and Feynman rules it is now straightforward 
to compute the loop induced FCNC rates for the decays (||) and (|]). We shall refrain 
from listing the lengthy analytical formulae as the computation is similar to the one 
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reported in great detail in Ref. [0. Therefore, we will limit ourselves to exhibit the final 
numerical results. The fiducial ratio on which we will apply our numerical computation 
is the following: 

for each Type j = I, II of 2HDM and for each neutral Higgs boson h = h^, H^, A^. While 
this ratio is not the total branching fraction, it is enough for most practical purposes and 
it is useful in order to compare with previous results in the literature. Notice that for 
mH± > trit (the most probable situation for Type II 2HDM's, see below) the ratio (|) 
reduces to B^{t h + c) = (t ^ h + c) /T(t ^ + 6), which is the one that we 
used in Ref. [0]. It is understood that P(t — > /i + c) above is computed from the 
one-loop diagrams in Fig. |l|, with all quark families summed up in the loop. Therefore, 
consistency in perturbation theory requires to compute r(t — + h) and T{t + h) 

in the denominator of only at the tree-level (for explicit expressions see e.g. 0). As 
mentioned in Sec. 2, we wish to compare our results for the Higgs channels (|^) with those 
for the gluon channel (0), so that we similarly define 

BHt^q + c) = VKt-^g + c) 

^ T{t^W+ + h) + T^{t^H+ + h)- ^> 

We have performed a fully-fledged independent analytical and numerical calculation of 
r-^(t — > -t- c) at one-loop in the context of 2IIDM I and II. Where there is overlapping, 
we have checked the numerical results of Ref. 0, but we point out that they agree with 
us only if r(t — >■ -\- b) is included in the denominator of eq. (P), in contrast to what 
is asserted in that reference in which B{t —>■ g + c) is defined without the charged Higgs 
channel contribution. 

We have performed part of the analytical calculation of the diagrams for both pro- 
cesses dl) and (H) by hand and we have cross-checked our results with the help of the 
numeric and algebraic programs FeynArts, FormCalc and LoopTools with which we 



have completed the rest of the calculation. In particular, the cancellation of UV diver- 
gences in the total amplitudes was also verified by hand. In addition we have checked 
explicitly the gauge invariance of the total analytical amplitude for the process (|^), which 
is a powerful test. And we have confirmed that our code reproduces the SUSY Higgs 
contribution of Ref. |n] when we turn on the MSSM Higgs mass relations. 



As mentioned above, a highly relevant parameter is tan/3, which must be restricted 
to the approximate range 

0.1 < tan/? < 60 (10) 

in perturbation theory^. It is to be expected from the various couplings involved in 
the processes under consideration that the low tan/3 region could be relevant for both 
the Type I and Type II 2HDM's. In contrast, the high tan/? region is only potentially 

^Some authors |Q claim that perturbativity allows tan/3 to reach values of order 100 and beyond, 
and these are still used in the literature. We consider it unrealistic and we shall not choose tan /? outside 



the interval (10). Plots versus tan/3, however, will indulge larger values just to exhibit the dramatic 



enhancements of our FCNC top quark rates. 
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important for the Type II. However, the eventually relevant regions of parameter space 
are also determined by the value of the mixing angle a, as we shall see below. 

Of course there are several restrictions that must be respected by our numerical anal- 
ysis. Above all the quadratic violations of SU{2) "custodial symmetry" must be within 
experimental bound. Therefore, the one-loop corrections to the p-parameter from the 
2HDM sector cannot deviate from the reference SM contribution in more than one per 
mil §: 

|5p2^^o^| ^0.001. (11) 

From the analytical expression for 6p in the general 2HDM we have introduced this 
numerical condition in our codes. Moreover, non-SUSY charged Higgs bosons from Type II 
models are subject to a very important indirect bound from radiative B-meson decays, 
specifically the experimental measurement by CLEO of the branching fraction BR{B — »• 



Xg'y) - equivalently BR(h — 57) at the quark level |2^. At present the data yield: 



BR{h ^57) = (3.15 ± 0.35 ± 0.32 ± 0.26) x 10"^ (12) 

The charged Higgs contribution to BR{h — > s 7) is positive; hence the larger is the exper- 
imental rate the smaller can be the charged Higgs boson mass. From the various analysis 
in the literature one finds mH± > (165 — 200) GeV for virtually any tan/5 > 1 |2^, p5| . 
This bound does not apply to Type I models because at large tan j3 the charged Higgs cou- 
plings are severely suppressed, whereas at low tan /? we recover the previous unrestricted 
situation of Type II models. Therefore, in principle the top quark decay t —>■ H^-\-b is still 
possible in 2HDM I; but also in 2HDM II, if mjj± lies near the lowest end of the previous 
bound, and in this case that decay can contribute to the denominator of eqs. (H)-(|^). 

One may also derive lower bounds to the neutral Higgs masses for these models p6| . 
For instance, one may use the Bjorken process e~^e~ Z + h° and the associated Higgs 
boson pair production e'^e^ — » h^{H^) + to obtain the following bounds in most of 
the parameter space: rriho + m^o > 100 GeV or > 150 GeV depending on whether we 
accept any value of tan P or we impose tan P > 1 respectively ||2^ . In each of these cases 



there is a light mass corner in parameter space both in the CP-even and in the CP-odd 



mass ranges around m^^o^o = 20 — 30 GeV [27]. Notwithstanding, as it is shown by the 



fit analysis of precision electroweak data in Ref. in the large tan/5 region a light 

is statistically correlated with a light H^, so that this situation is not favored by the 
aforementioned bound from b ^ sj. Moreover, since our interest in Type II models is 
mainly focused in the large tan (3 regime, the corner in the light CP-even mass range is a 
bit contrived. At the end of the day one finds that, even in the worst situation, the strict 
experimental limits still allow generic 2HDM neutral scalar bosons as light as 70 GeV or 
so. As we said, most of these limits apply to Type II 2HDM's, but we will conservatively 
apply them to Type I models as well. 

Finally, for both models we have imposed the condition that the (absolute value) of 
the trilinear Higgs self-couplings do not exceed the maximum unitarity limit tolerated for 
the SM trilinear coupling: 



^hhh\ ^ 



X'^i^kmH = lTeV) = ^'\\T' (13) 



^g{lTeVf 
2Mw 

The combined set of independent conditions turns out to be quite effective in narrowing 
down the permitted region in the parameter space, as can be seen in Figs. where we 
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Figure 2: Evolution of the FCNC top quark fiducial ratios (§)-(§) in Type I 2HDM versus: 
(a) the mixing angle a in the CP-even Higgs sector, in units of vr; (b) tan j3. The values 
of the fixed parameters are as in eqs. (p!4D and (p!5|). 



plot the fiducial FCNC rates (^-(^ versus the parameters (^. The cuts in some of these 
curves just reflect the fact that at least one of these conditions is not fulfilled. 

After scanning the parameter space, we see in Figs. 0-^ that the 2HDM I (resp. 
2HDM II) prefers low values (resp. high values) of tana and tan/3 for a given chan- 
nel, e.g. if: — s> c. Therefore, the following choice of mixing angles will be made to 
optimize the presentation of our numerical results: 

2HDM I : tana = tan/5 = 1/4 ; 

2HDM II : tana = tan/? = 50 . (14) 

We point out that, for the same values of the masses, one obtains the same maximal FCNC 
rates for the alternative channel t c provided one just substitutes a 7r/2 — a. 

Equations (|T^ define the eventually relevant regions of parameter space and, as mentioned 
above, depend on the values of the mixing angles a and (3, namely /? ~ a ~ for Type I 
and /? ~ a ~ 7r/2 for Type II. 

Due to the a 7r/2 — a symmetry of the maximal rates for the CP-even Higgs 
channels, it is enough to concentrate the numerical analysis on one of them, but one has 
to keep in mind that the other channel yields the same rate in another region of parameter 
space. Whenever a mass has to be fixed, we choose conservatively the following values 
for both models: 

= 100 GeV , m^o = 150 GeV , niAo = mH± = 180 GeV . (15) 

Also for definiteness, we take the following values for some relevant SM parameters in our 
calculation: 

mt = 175 GeV , rub = 5 GeV , a,(mt) = 0.11 , Kb = 0.040 , (16) 

and the remaining ones are as in [Q. Notice that our choice of m^o prevents the decay 
t ^ c from occurring, and this is the reason why it does not appear in Figs. 0- 
|. The variation of the results with respect to the masses is studied in Figs. In 
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Figure 3: As in Fig. but for the 2HDM II. The plot in (b) continues above the bound 
in eq. (|TD|) just to better show the general trend. 

particular, in Fig. ^ we can see the (scanty) rate of the channel t c when it is 

kinematically allowed. In general the pseudoscalar channel is the one giving the skimpiest 
FCNC rate. This is easily understood as it is the only one that does not have trilinear 
couplings with the other Higgs particles (Cf. Table |l]). While it does have trilinear 
couplings involving Goldstone bosons, these are not enhanced. The crucial role played by 
the trilinear Higgs self-couplings in our analysis cannot be underestimated as they can 
be enhanced by playing around with both (large or small) tan/? and also with the mass 
splittings among Higgses. This feature is particularly clear in Fig. ^ where the rate of 
the channel t ^ c is dramatically increased at large m^o, for fixed values of the other 
parameters and preserving our list of constraints. Similarly would happen for t c 
in the corresponding region a — 7r/2 — a. 

From Figs. 0a and 0b it is pretty clear that the possibility to see FCNC decays of 
the top quark into Type I Higgs bosons is plainly hopeless even in the most favorable 
regions of parameter space - the lowest (allowed) tan/3 end. In fact, the highest rates 
remain neatly down 10~^, and therefore they are (at least) one order of magnitude below 
the threshold sensibility of the best high luminosity top quark factory in the foreseeable 
future (see Section 4). We remark, in Fig. ^ that the rate for the reference decay t ^ gcin 
the 2HDM I is also too small but remains always above the Higgs boson rates. Moreover, 
for large tan P one has, as expected, { t ^ g c) ^ B^^^ {t^gc)c::iAx 10~^^ because 
in this limit all of the charged Higgs couplings in the 2HDM I (the only Higgs couplings 
involved in this decay) drop off. Due to the petty numerical yield from Type I models we 
refrain from showing the dependence of the FCNC rates on the remaining parameters. 

Fortunately, the meager situation just described does not replicate for Type II Higgs 
bosons. For, as shown in Figs. |^ and ^b, the highest potential rates are of order 10~^, 
and so there is hope for being visible. In this case the most favorable region of parameter 
space is the high tan (3 end in eq. (|Tnp. Remarkably, there is no need of risking values over 
and around 100 (which, as mentioned above, are sometimes still claimed as perturbative!) 
to obtain the desired rates. But it certainly requires to resort to models whose hallmark is 
a large value of tan/3 of order or above mt/mb ^ 35. As for the dependence of the FCNC 
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Figure 4: Evolution of the FCNC top quark fiducial ratios (D-(|D in Type II 2HDM 
versus: (a) the CP-odd Higgs boson mass m^o; (b) the charged Higgs boson mass mH±- 
The values of the fixed parameters are as in eqs. and (|T5|). The plot in (b) starts 
below the bound uih^ > 165 GeV mentioned in the text to better show the general trend. 

rates on the various Higgs boson masses (Cf. Figs. we see that for large m^o the decay 
t ^ c can be greatly enhanced as compared to t ^ gc; and of course, once again, the 
same happens with t — » c in the alternative region a 7r/2 — a. We also note (from 
the combined use of Figs, ^o, ^ and ^b) that in the narrow range where t b could 

still be open in the 2HDM II, the rate oi t c becomes the more visible the larger 

and larger is tan/5 and m^o. Indeed, in this region one may even overshoot the 10~^ level 
without exceeding the upper bound (|l^) while also keeping under control the remaining 
constraints, in particular eq. (pAj). Finally, the evolution of the rates (@)-(^ with respect 
to the two CP-even Higgs boson masses is shown in Figs. |^a and The neutral Higgs 
bosons themselves do not circulate in the loops (Cf. Fig. |I]) but do participate in the 
trilinear couplings (Cf. Table |lD and so the evolution shown in some of the curves in 
Fig. 1^ is due to both the trilinear couplings and to the phase space exhaustion. 

Turning now to the light scalar and pseudoscalar corners in parameter space mentioned 
above, it so happens that, after all, they prove to be of little practical interest in our case. 
Ultimately this is due to the quadratic Higgs boson mass differences entering 5p which 
make very difficult to satisfy the bound (pT]) . The reason being that for Type II models the 
limit mH± ~ 165 GeK from b —>■ s'-f implies that the constraint ( pT] ) cannot be preserved 
in the presence of light neutral Higgses. In actual fact the analysis shows that if e.g. 
one fixes rriho = 20 — 30 GeV, then the minimum m^o allowed by 6p is 100 GeV^ and 
the maximum rate (^ is of order 10~^. Conversely, if one chooses m^o = 20 — 30 GeV, 
then the minimum m/jo allowed by Sp is 120 GeV and the maximum rate (^ is near 10"*^. 
Although in the last case the maximum rate is higher than in the first case, it is just of the 
order of the maximum rate already obtained outside the light mass corners of parameter 
space. On the other hand, these light mass regions do not help us in Type I models either. 
Even though for these models we do not have the b —>■ sj bound on the charged Higgs, we 



still have the direct LEP 200 bound mf{± ~ 78.7 GeV [28] which is of course weaker than 



the CLEO bound. As a consequence the 6p constraint can be satisfied in the 2HDM I for 
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Figure 5: As in Fig. ^, but plotting versus: (a) the lightest CP-even Higgs boson mass 
rriho; (b) the heaviest CP-even Higgs boson mass mj^o. 

neutral Higgs bosons lighter than in the corresponding 2HDM II case, and one does get 
some enhancement of the FCNC rates. Specifically, one may reach up to 10^^. However, 
the maximal rates (P) for the 2HDM I Higgs bosons are so small (see Figs. Qa-]^) that 
this order of magnitude enhancement is rendered immaterial. The upshot is that the top 
quark FCNC processes are not especially sensitive to the potential existence of a very 
light Higgs boson in either type of 2HDM. 

4 Discussion and conclusions 

The sensitivities to FCNC top quark decays for 100 fb~^ of integrated luminosity in the 
relevant colliders are estimated to be p9[ : 



LHC -.Bit cX) > 5 X 10-^ 
LC:B{t^cX)>5xlO'\ (17) 
TEV33 -.Bit ^ cX) > 5 X 10"^ . 

This estimation has been confirmed by a full signal-background analysis for the hadron 
colliders in Ref . [0O[] . From these experimental expectations and our numerical results 
it becomes patent that whilst the Tevatron will remain essentially blind to this kind of 
physics, the LHC and the LC will have a significant potential to observe FCNC decays of 
the top quark beyond the SM. Above all there is a possibility to pin down top quark decays 
into neutral Higgs particles, eq. (0), within the framework of the general 2HDM II provided 
tan/3 ^ mt/rrib ~ 35. The maximum rates are of order 10~^ and correspond to the two 
CP-even scalars. This conclusion is remarkable from the practical (quantitative) point of 
view, and also qualitatively because the top quark decay into the SM Higgs particle is, in 
notorious contradistinction to the 2HDM II case, the less favorable top quark FCNC rate 
in the SM. On the other hand, we deem practically hopeless to see FCNC decays of the 
top quark in a general 2HDM I for which the maximum rates are of order 10"''. This order 
of magnitude cannot be enhanced unless one allows tan/5 <^ 0.1, but the latter possibility 
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is unrealistic because perturbation theory breaks down and therefore one cannot make 
any prediction within our approach. 

We have made a parallel numerical analysis of the gluon channel t ^ c g in both types 
of 2HDM's. We confirm that this is another potentially important FCNC mode of the top 
quark in 2HDM extensions of the SM |Q but, unfortunately, it still falls a bit too short 
to be detectable. The maximum rates for this channel lie below 10~^ in the 2HDM I (for 
tan/3 > 0.1) and in the 2HDM II (for tan/? < 60), and so it will be hard to deal with it 
even at the LHC 

We are thus led to the conclusion that the Higgs channels (^, more specifically the 
CP-even ones, give the highest potential rates for top quark FCNC decays in a general 
2HDM II. Most significant of all: they are the only FCNC decay modes of the top quark, 
within the simplest renormalizable extensions of the SM, that have a real chance to be 
seen in the next generation of high energy, high luminosity, colliders. 



The former conclusions are similar to the ones derived in Ref. \T^] for the MSSM 
case, but there are some conspicuous differences on which we wish to elaborate a bit 
in what follows First, in the general 2HDM II the two channels t — > (/i°,i^°) c 

give the same maximum rates, provided we look at different (disjoint) regions of the 
parameter space. The t ^ c channel is, as mentioned, negligible with respect to the 
CP-even modes. Hereafter we will discard this FCNC top quark decay mode from our 
discussions within the 2HDM context. On the other hand, in the MSSM there is a most 
distinguished channel, viz. t —>■ c, which can be high-powered by the SUSY stuff all 
over the parameter space. In this framework the mixing angle a becomes stuck once tan/3 
and the rest of the independent parameters are given, and so there is no possibility to 
reconvert the couplings between and as in the 2HDM. Still, we must emphasize 
that in the MSSM the other two decays t — c and t ^ c can be competitive 
with t c in certain portions of parameter space. For example, t H c becomes 



competitive when the pseudoscalar mass is in the range 110 GeV^ < m^o < 170 GeV 



The possibility of having more than one FCNC decay (g) near the visible level is a feature 
which is virtually impossible in the 2HDM II. Second, the reason why t — >^ /i" c in the 
MSSM is so especial is that it is the only FCNC top quark decay (0) which is always 
kinematically open throughout the whole MSSM parameter space, while in the 2HDM 
all of the decays (0) could be, in the worse possible situation, dead closed. Nevertheless, 
this is not the most likely situation in view of the fact that all hints from high precision 
electroweak data seem to recommend the existence of (at least) one relatively light Higgs 



boson [^, This is certainly an additional motivation for our work, as it leads us to 
believe that in all possible (renormalizable) frameworks beyond the SM, and not only in 
SUSY, we should expect that at least one FCNC decay channel (|^) could be accessible. 
Third, the main origin of the maximum FCNC rates in the MSSM traces back to the 
tree-level FCNC couplings of the gluino WM. These are strong couplings, and moreover 



they are very weakly restrained by experiment. In the absence of such gluino couplings, or 
perhaps by further experimental constraining of them in the future, the FCNC rates in the 
MSSM would boil down to just the electroweak (EW) contributions, to wit, those induced 
by charginos, squarks and also from SUSY Higgses. The associated SUSY-EW rate is of 
order 10~^ at most, and therefore it is barely visible, most likely hopeless even for the 
LHC. In contrast, in the general 2HDM the origin of the contributions is purely EW and 
the maximum rates are two orders of magnitude higher than the full SUSY-EW effects in 
the MSSM. It means that we could find ourselves in the following situation. Suppose that 
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the FCNC couplings of the gluino get severely restrained in the future and that we come 
to observe a few FCNC decays of the top quark into Higgs bosons, perhaps at the LHC 
and/or the LC. Then we would immediately conclude that these Higgs bosons could not 
be SUSY-MSSM, whilst they could perhaps be CP-even members of a 2HDM II. Fourth, 
the gluino effects are basically insensitive to tan/?, implying that the maximum MSSM 
rates are achieved equally well for low, intermediate or high values of tan /?, whereas the 
maximum 2IIDM II rates (comparable to the MSSM ones) are attained only for high 
tan p. 

The last point brings about the following question: what could we possibly conclude if 
the gluino FCNC couplings were not further restricted by experiment and the tagging of 
certain FCNC decays of the top quark into Higgs bosons would come into effect? Would 
still be possibly to discern whether the Higgs bosons are supersymmetric or not? The 
answer is, most likely yes, provided certain additional conditions would be met. 

There are many possibilities and corresponding strategies, but we will limit ourselves 
to point out some of them. For example, let us consider the type of signatures involved 
in the tagging of the Higgs channels. In the favorite FCNC region (0) of the 2HDM II, 
the combined decay t ^ h c ^ ebb is possible only for or for but not for both - 
Cf. Fig. ^ - whereas in the MSSM, together with H^, are highlighted for 110 GeV < 
rriAo < rrit, with no preferred tan (3 value. And similarly, t — > A*^ c is also non-negligible for 
rriAO ~ 120 GeV |T^. Then the process t —>■ h e —>■ ebb gives rise to high charm-quark 
jets and a recoiling bb pair with large invariant mass. It follows that if more than one 
distinctive signature of this kind would be observed, the origin of the hypothetical Higgs 
particles could not probably be traced back to a 2HDM II. 

One might worry that in the case of and they could also (in principle) decay 
into electroweak gauge boson pairs h^,H^ VewVew, which in some cases could be 
kinematically possible. But this is not so in practice for the 2HDM II if we stick to 
our favorite scenario, eq. (|I^). In fact, we recall that the decay VewVew is not 

depressed with respect to the SM Higgs boson case provided P — a = n/2, and similarly 
for —>■ VewVew if /3 — a = 0. However, neither of these situations is really pinpointed 
by FCNC physics because we have found (3 ~ 7r/2 in the most favorable region of our 
numerical analysis, and moreover a was also seen there to be either a ^ 7r/2 (for h^) or 
(for H'^), so both decays VewVew are suppressed in the regions where the FCNC 

rates of the parent decays t — >■ {h^, H^) e are maximized. Again, at variance with this 
situation, in the MSSM case — >• VewVew is perfectly possible - not so VewVew 
due to the aforementioned upper bound on m/jo - because tan (3 has no preferred value in 
the most favorable MSSM decay region of t ^ e. Therefore, detection of a high pt 
charm-quark jet against a VewVew pair of large invariant mass could only be advantageous 
in the MSSM, not in the 2HDM. Similarly, for tan /3 > 1 the decay H° h° (with real 
or virtual h^) is competitive in the MSSM in a region where the parent FCNC top 
quark decay is also sizeable. Again this is impossible in the 2HDM II and therefore it can 
be used to distinguish the two (SUSY and non-SUSY) Higgs frames. 

Finally, even if we place ourselves in the high tan (3 region both for the MSSM and the 
2HDM II, then the two frameworks could still possibly be separated provided that two 
Higgs masses were known, perhaps one or both of them being determined from the tagged 
Higgs decays themselves, eq. (0). Suppose that tan/5 is numerically known (from other 
processes or from some favorable fit to precision data), then the full spectrum of MSSM 
Higgs bosons would be approximately determined (at the tree level) by only knowing 
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one Higgs mass, a fact that could be used to check whether the other measured Higgs 
mass becomes correctly predicted. Of course, the radiative corrections to the MSSM 
Higgs mass relations can be important at high tan/? pO[, but these could be taken into 



account from the approximate knowledge of the relevant sparticle masses obtained from 
the best fits available to the precision measurements within the MSSM. If there were 
significant departures between the predicted mass for the other Higgs and the measured 
one, we would probably suspect that the tagged FCNC decays into Higgs bosons should 
correspond to a non-supersymmetric 2HDM II. 

At the end of the day we see that even though the maximum FCNC rates for the MSSM 
and the 2HDM II are both of order 10~^ - and therefore potentially visible - at some point 
on the road it should be possible to disentangle the nature of the Higgs model behind 



the FCNC decays of the top quark. Needless to say, if all the recent fuss at CERN p8 
about the possible detection of a Higgs boson would eventually be confirmed, this could 
still be interpreted as the discovery of one neutral member of an extended Higgs model. 
Obviously the combined Higgs data from LEP 200 and the possible discovery of FCNC 
top quark decays into Higgs bosons at the LHC/LC would be an invaluable cross-check 
of the purportedly new phenomenology. 

We emphasize our most essential conclusions in a nutshell: i) Detection of FCNC top 
quark decay channels into a neutral Higgs boson would be a blazing signal of physics 
beyond the SM; ii) There is a real chance for seeing rare events of that sort both in 
generic Type II 2HDM's and in the MSSM. The maximum rates for the leading FCNC 
processes (|]) and (|^) in the 2HDM II (resp. in the MSSM) satisfy the relations 

BR{t -^gc) < 10-^(10-^) < BR{t ^hc)^ 10"^ , (18) 

where it is understood that h is or but not both, in the 2HDM II; whereas h is 
most likely h^, but it could also be and A", in the MSSM ; iii) Detection of more than 
one Higgs channel would greatly help to unravel the type of underlying Higgs model. 

The pathway to seeing new physics through FCNC decays of the top quark is thus 
potentially open. It is now an experimental challenge to accomplish this program using 
the high luminosity super-colliders round the corner. 
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